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ABSTRACT 


Technological  advancements,  especially  in  communications  systems, 
have  led  to  a  burgeoning  interest  in  network  centric  warfare  (NCW), 
fundamentally  changing  how  warfare  is  being  conducted.  Network  centric  warfare 
(NCW)  systems  are  being  rushed  to  the  field  and  are  offered  as  a  solution  for  the 
‘fog  of  war’  and  as  a  way  to  reduce  manpower  costs.  To  date,  there  are  no 
empirical  findings  that  support  or  refute  these  NCW  system  claims. 

The  goal  of  this  thesis  was  to  ascertain  the  utility  of  the  Geographical 
Recall  and  Analysis  of  Data  in  the  Environment  (GRADE)  as  a  method  and 
process  by  which  complex  human-technological  systems  can  be  assessed.  The 
GRADE  builds  upon  the  Dynamic  Model  of  Situated  Cognition  (DMSC).  This 
study  essentially  determines  if  GRADE  could  be  used  in  model  validation  in 
laboratory  and  field  settings  for  evaluating  NCW  claims.  Unfortunately,  that 
research  goal  was  not  entirely  realized  due  to  constraints  and  limitations  in  the 
data  collection  exercise. 

The  thesis  discusses  the  ‘lessons  learned’  from  this  research  effort  and 
makes  recommendations  about  future  exercises  and  how  to  better  populate  the 
DMSC  with  data.  Additional  recommendations  for  changes  to  the  processes  and 
procedures  for  data  collection  are  provided. 
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I.  INTRODUCTION 


A.  PREFACE 

Accurate  and  timely  information  is  critical  to  success  in  battle.  Throughout 
history,  commanders  and  foot  soldiers  have  collected  and  transmitted  battle 
information  in  various  methods.  These  methods  have  included  fire  and  smoke 
signals,  homing  pigeons,  and  runners.  During  WWII,  new  methods  to  exchange 
information  included  telephones,  telegraph,  and  wireless  radios.  The  twenty-first 
century  has  brought  us  fully  into  the  digital  age  in  which  we  transfer  information 
to  distributed  locations  over  the  globe  by  using  wireless  networks,  the  internet, 
and  voice  over  IP  (VOIP). 

These  technological  advancements  in  communication  cause  a  change  in 
how  warfare  is  conducted.  Network  centric  warfare  (NOW)  systems  are  being 
rushed  to  the  field  offered  as  a  solution  for  the  ‘fog  of  war’  and  as  a  way  to 
reduce  manpower  costs.  Many  of  the  new  technologies  that  enable  NOW  have 
given  little  consideration  to  the  role  of  the  human  in  the  design  or  implementation 
of  the  system.  There  is  an  unstated  and  untested  assumption  that  system 
performance  is  facilitated  by  the  introduction  of  any  new  piece  of  equipment. 
This  thesis  will  describe  a  method  to  test  these  assumptions  by  objectively 
assessing  system  performance  using  a  process  tracing  approach.  This  proposed 
method  will  provide  NCW  system  developers  with  the  ability  to  collect  objective 
data  on  the  total  system’s  contribution  to  a  shared  understanding  of  the 
battlefield  and  provide  decision  makers  with  information  on  the  limitations  and 
capabilities  of  this  technology. 

This  introductory  chapter  will  first  provide  an  overview  of  network  centric 
warfare.  Then  a  definition  of  Human  System  Integration  (HSI)  is  provided  to 
explain  the  process  by  which  the  individual  domains  of  HSI  are  integrated  in  a 
human-centered  design  process.  Lastly,  three  of  the  eight  domains  of  HSI  will  be 
further  defined  within  the  context  of  NCW  for  the  purpose  of  this  thesis. 
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B.  NETWORK  CENTRIC  WARFARE 

Network  centric  warfare  (NCW)  is  based  on  a  concept  of  operations  in 
which  humans  use  technology  to  share  information  and  the  ability  to  rapidly 
utilize  that  information  will  result  in  increased  military  effectiveness.  The  goal  of 
NCW  is  to  increase  combat  power  by  utilizing  all  assets  in  the  field  and  at  home 
to  electronically  link  decision-makers  and  shooters.  With  access  to  the  same 
information  at  the  same  time,  NCW  theory  suggests  that  this  new  technology  will 
enable  operators  to  achieve  shared  awareness  resulting  in  increased  speed  of 
command,  higher  tempo  of  operations,  greater  lethality,  increased  survivability 
and  better  synchronization  (Alberts  &  Garstka,  &  Stein,  1999;  Alberts  &  Hayes, 
2003;  Cebrowski  &  Garstka,  1998). 

The  Department  of  Defense  (DoD)  is  currently  developing  and  fielding 
numerous  versions  of  this  networked  technology,  but  the  goals  of  NCW  cannot 
be  achieved  with  technology  alone.  NCW  is  an  emerging  theory  of  war  and  a 
new  concept  of  operations,  and  it  provides  an  opportunity  to  develop  a  human- 
centered  system  of  systems.  Network  centric  warfare  requires  a  cultural  change 
away  from  traditional  hierarchical  military  relationships  to  include  the  new  ability 
of  networking  over  the  Internet  among  large  groups  of  people  to  accomplish  a 
combat  objective  in  real  time.  The  technology  itself  will  not  be  an  optimal  force 
multiplier  unless  humans  are  recognized  as  critical  information  nodes  and 
sensors  within  the  system. 

In  NCW  the  humans  are  vital  to  the  system  because  they  are  required  to 
define,  forage  and  harvest  the  information  needed  to  accomplish  an  objective. 
The  goal  then  becomes  developing  a  system  that  is  both  transparent  and  intuitive 
to  the  operator  while  providing  critical  information  to  decision  makers  in  a  timely 
manner.  To  this  end,  if  a  human-centered  design  is  used  in  the  development 
process,  the  cost  for  the  entire  life  cycle  of  the  NCW  system  may  be  reduced, 
allowing  the  overall  system  to  operate  more  effectively. 
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C.  HUMAN  SYSTEMS  INTEGRATION 

Human  Systems  Integration  (HSI)  is  a  human-centered  approach  to  the 
design,  development,  and  acquisition  of  systems  from  cradle  to  grave.  A 
fundamental  premise  of  HSI  is  that  any  new  materiel  solution  must  consider  the 
humans  involved  in  the  system.  The  acquisition  process  should  include  human 
considerations  early  in  the  design,  thereby  enhancing  human-system 
performance.  Implementing  HSI  early  in  the  acquisition  process  can  result  in  a 
reduction  of  total  lifecycle  costs,  and  better  overall  system  performance. 

To  achieve  a  human-centered  design,  HSI  professionals  provide  a 
comprehensive  management  and  technical  strategy  for  human  systems 
integration.  This  HSI  plan  is  a  DoD  requirement  as  stated  in  the  5000.2  series 
instruction  document  (USD/AT&L,  2002).  While  there  is  considerable  debate 
about  the  number  and  names  of  the  HSI  domains,  the  Naval  Postgraduate 
School,  which  offers  the  only  graduate  degree  in  HSI,  uses  eight  domains  in  its 
definition  of  HSI.  These  eight  domains  are:  Human  Factors  Engineering, 
Manpower,  Personnel,  Training,  Human  Survivability,  Health  Hazards,  System 
Safety,  and  Habitability.  Following  is  a  description  of  each  of  these  eight  HSI 
domains. 

1.  Training 

The  instruction,  education,  and  training  required  to  provide  personnel  with 
the  knowledge,  skills  and  abilities  needed  to  operate  and  maintain  systems 
(NPS,  2006).  For  example,  on-the-job  training  is  an  organization’s  attempt  to 
assist  task  specific  learning  through  instruction,  observation,  or  practice  that 
reflects  the  demands  of  a  job. 

2.  Personnel 

The  knowledge,  skills,  abilities,  aptitudes,  and  experience  required  for  the 
human  component  of  the  system  to  enable  the  system  to  achieve  full  capability 
(NPS,  2006).  Personnel  selection  is  based  upon  the  tasks  that  are  to  be 
performed  to  complete  a  job.  For  example,  operators,  maintainers,  and  support 
personnel  would  have  different  KSAs  and  experience. 
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3.  Human  Factors  Engineering 

A  design  and  development  process  that  focuses  on  the  interaction 
between  human  beings  and  the  tools,  equipment,  machines  and  systems  they 
employ  to  achieve  system  effectiveness  and  safety  (NPS,  2006).  There  are  two 
objectives  within  a  human-centered  design:  consider  ergonomic  design  (the 
hardware)  and  create  an  intuitive  and  transparent  system  (software)  for  the 
operator. 

4.  Manpower 

The  total  demand,  expressed  by  the  number  of  individuals,  required  to  fill 
jobs,  slots,  or  billets  for  a  system  with  consideration  for  total  life  cycle 
requirements  to  include  the  big  areas  of  both  operational  and  support  (NPS, 
2006).  For  example,  manpower  would  represent  the  number  of  bodies  required 
to  supervise,  train,  maintain  and  operate  an  aircraft,  but  does  not  represent  KSAs 
required  for  each  billet. 

5.  System  Safety 

System  safety  is  the  application  of  engineering,  management,  and  HSI 
principles,  as  well  as  human  factors  design  criteria,  and  techniques  to  achieve 
acceptable  levels  of  risk  hazard  for  mission  requirements  and  system  capability 
(NPS,  2006).  For  example,  a  weapon  system  is  designed  to  reduce  the  chance 
of  death  or  injury  to  operators  within  the  constraints  of  operational  effectiveness, 
cost,  schedule,  and  performance  throughout  all  phases  of  the  system  life  cycle. 

6.  Human  Survivability 

Human  survivability  is  the  ability  of  a  system  and  its  personnel  to  avoid  or 
withstand  hostile  environments  without  jeopardizing  mission  accomplishment 
(NPS,  2006).  For  example,  in  the  case  of  a  tank,  it  is  the  ability  of  the  crew  and 
the  overall  system  to  continue  to  function  during  and  after  a  disturbance  from 
something  as  simple  as  a  rollover  due  to  terrain  or  as  complex  as  withstanding  a 
direct  hit  from  an  armor  piercing  round. 

7.  Health  Hazards 

Health  hazards  are  anything  that  can  have  a  harmful  effect  on  health 
either  during  the  system’s  production,  operation,  or  maintenance  to  include 
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decreased  worker  performance  and  significant  long-term  health  risks  (NPS, 
2006).  For  example,  health  hazard  specialists  may  be  concerned  with  the 
toxicity/lethality  of  jet  fuel  and  the  personal  protection  equipment  (PPE)  required 
for  personnel  to  work  in  and  repair  aircraft  fuel  cells. 

8.  Habitability 

Habitability  is  concerned  with  the  qualities  of  the  physical  living 
environment  and  support  services  within  a  system  which  lead  to  mission 
effectiveness  (NPS,  2006).  Using  a  space  station  as  an  example,  it  is  important 
that  the  crew  resting  quarters  are  away  from  heat  and  noise  (physical  stressors), 
the  work  area  has  adequate  space  to  prevent  overcrowding  (psychological 
stressors),  and  that  there  are  adequate  facilities  for  food  preparation,  bathing  and 
waste  removal.  These  accommodations  will  contribute  greatly  to  the 
performance  of  the  crew  and  the  total  system. 

D.  THE  RELATIONSHIP  BETWEEN  NCW  AND  HSI 

This  thesis  uses  a  human-centered  approach  that  encompasses  the 
following  three  domains  of  HSI:  training,  personnel,  and  human  factors 
engineering.  These  three  domains  will  now  be  defined  within  the  context  of 
NCW. 

1.  Training 

NCW  systems  require  a  transition  from  legacy  systems  and  concepts  of 
operations  to  information  age  weapons,  equipment,  and  technology  (Money, 
2001).  This  transition  also  requires  new  concepts  of  operations  that  facilitate 
new  strategies  and  tactics  by  providing  training  for  decentralized  rapid  operations 
for  use  in  asymmetrical  warfare.  NCW  systems  require  that  operators  receive 
training  in  both  computer/digital  skills  and  in  communication  and  coordination 
within  horizontal  and  vertical  networks. 

2.  Personnel 

To  reduce  training  time  and  cost  for  NCW  systems,  it  is  imperative  to 
select  and  place  the  correct  personnel  in  the  right  jobs  (Money,  2001).  For  the 
NCW  system  to  perform  optimally,  personnel  will  need  to  have  information 
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technology  skills,  an  aptitude  for  multitasking,  and  experience  in  a  position  that 
requires  the  individual  to  fulfill  many  roles  and  duties  simultaneously. 

3.  Human  Factors  Engineering 

NCW  systems  need  to  have  a  human-centered  design  to  maximize  their 
potential  (Money,  2001).  For  example,  designing  computer  displays  that  support 
situational  awareness  and  control  of  real-time  and  dynamic  processes  for  NCW 
means  more  than  just  presenting  the  data  in  a  form  that  supports  the  workflow  of 
the  operator.  To  enable  effective  understanding  of  the  battlespace,  it  is 
necessary  to  portray  the  information  in  a  manner  that  supports  the  cognitive 
capability  and  goals  of  the  operator.  HFE  is  critical  in  all  cases  where  the 
operator  is  performing  at  the  extremes  of  his  or  her  performance  envelopes. 
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II.  LITERATURE  REVIEW 


This  chapter  reviews  literature  from  the  fields  of  network  centric  warfare 
(NCW),  team  performance,  development  of  virtual  teams,  and  situated  cognition. 

A.  NETWORK  CENTRIC  WARFARE 

The  current  direction  of  the  DoD  has  shifted  from  platform-centric  warfare 
to  NCW  or  network  centric  operations  (NCO)  (Cebrowski  &  Garstka,  1998).  The 
potential  of  information  age  warfare  was  first  recognized  in  the  1983  invasion  of 
Grenada,  reinforced  in  the  1990  Persian  Gulf  War,  and  widely  accepted  in  the 
2003  invasion  of  Iraq  and  the  war  on  terror  in  Afghanistan  (Wilson,  2004). 
Networked  information  technologies  supplied  the  forces  with  unprecedented 
battlefield  advantages  during  these  wars.  These  information  technologies  are 
capable  of  providing  leadership  with  real-time  awareness  of  current  theaters. 
Reportedly,  operators  in  the  field  who  use  NCW  systems  are  also  experiencing 
increased  situational  awareness  on  the  battlefield  (Garstka,  2003;  Blash,  2003). 
The  current  Department  of  Defense  trend  is  to  reshape  the  US  military  as  a 
network  centric  force  that  will  be  able  to  leverage  information  to  increase  speed 
and  combat  effectiveness. 

NCW  combines  various  types  of  information  technologies  to  produce 
information  superiority  that  is  reputed  to  increase  combat  power  through  self¬ 
synchronization  and  other  network  centric  operations  (Alberts,  1999;  Alberts  & 
Garstka,  2001;  Wilson  2004).  A  robust,  secure  and  broadband  network  plays  a 
crucial  role  in  future  warfare  in  which  the  network  will  connect  all  types  of  sensors 
in  the  air,  on  land,  and  under  the  sea.  Ideally,  the  network  will  assist  with 
logistics  to  ensure  the  timely  supply  of  troops  and  help  synchronize  conventional, 
electronic,  and  information  attacks. 

B.  TEAM  PERFORMANCE 

Outcomes  in  time-critical,  high-risk  situations  are  often  dependent  on 
crisis  action  teams  or  “tiger  teams”  in  which  expertise,  information,  and  tasks  are 
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distributed  across  highly  trained  and  specialized  individuals.  The  effectiveness  of 
these  teams  requires  rapid,  complex,  coordinated  behavior,  resulting  in 
integrated  team  performance.  The  high  degree  of  interdependence  that  is 
inherent  in  such  teams  makes  the  performance  requirements  of  individual 
members  particularly  demanding  (Kozlowski,  Gully,  McHugh,  Salas,  Cannon- 
Bowers,  1996a).  Previous  research  indicates  that  the  contribution  of  team 
members  is  not  simply  an  additive  function  of  each  individual.  Rather,  the 
patterns  of  communication,  information  flow,  and  responses  which  indicate  a 
shared  mental  model  of  team  members  enables  the  team  to  respond  to 
externally  driven  and  constantly  changing  task  demands  (Thordsen  &  Klein, 
1989;  Fleishman  &  Zaccaro,  1992;  Weick  &  Roberts,  1993).  To  achieve  high 
team  effectiveness,  it  is  believed  that  teamwork-building  or  enabling  skills  may  be 
needed  to  facilitate  a  congruent  and  synchronous  mental  model  of  action  for 
team  members  (Kleinman  &  Serfaty,  1989). 

Thus  far,  research  in  the  field  of  team  performance  has  led  to  an 
increasing  focus  on  work  teams  within  organizations  and  on  how  best  to  build 
teamwork  skills  that  are  the  foundation  of  team  effectiveness  (Kozlowski,  Gully, 
McHugh,  Salas,  Cannon-Bowers,  1996b).  The  literature  suggests  that  any  team, 
including  a  command  and  control  team,  requires  a  certain  amount  of  time  to 
become  effective.  However,  the  opportunity  to  spend  time  together  as  a  team  is 
not  always  available.  This  can  be  seen  in  real  world  emergencies  where 
effective  ad-hoc  teams  come  together  in  a  short  period  of  time.  For  example, 
teams  formed  quickly  for  Hurricane  Katrina  emergency  care.  And  in  NCW  a 
team  may  be  formed  at  any  time  from  members  distributed  around  the  globe 
(Kayes,  2003). 

In  the  scientific  literature,  teams  are  distinguished  from  other  collectives  by 
the  characteristic  of  shared  interdependent  work  (Thordsen  &  Klein,  1989;  Salas, 
Dickinson,  Converse,  &  Tannenbaum,  1992).  In  comparison,  terms  “group”  or 
“work  group”  include  work  collectives  who  see  themselves  and  are  seen  by 
others  as  social  entities  with  common  goals  but  with  looser  task  connections 
(Thordsen  &  Klein,  1989). 
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Several  theories  describe  the  stages  teams  pass  through  as  they  evolve 
(Bennis  &  Nanus,  1985;  Bennis  &  Shepard,  1956;  Bion,  1961;  Schultz,  1958; 
Tuckman,  1965;  Tuckman  &  Jenson,  1977).  These  models  provide  a  description 
of  developmental  stages  of  teams  and  suggest  that  team  development  is  a 
process  that  occurs  over  time.  The  models  further  indicate  that  much  of  the  time 
required  to  develop  the  team  is  spent  on  social  rules  and  norms  that  reduce 
interpersonal  conflict  while  increasing  the  productivity  of  the  team.  For  example, 
Tuckman’s  (1965)  model  consists  of  the  following  four  stages:  forming,  storming, 
norming,  and  performing.  Although  this  model  is  a  stage  model,  it  does  not  imply 
a  progressive  directionality  to  a  team’s  ability  to  pass  through  the  stages.  In 
other  words,  a  team  can  revisit  any  of  the  stages,  for  example,  going  from 
forming  to  storming  and  then  back  to  forming  again.  These,  stage  models 
suggest  a  time-consuming  process  to  develop  an  effective  team.  Although  the 
issues  mentioned  are  one  aspect  of  team  development,  the  models  are  not  able 
to  explain  rapid  ad-hoc  team  formation  during  time-critical  events  such  as  military 
operations. 

C.  VIRTUAL  TEAMS 

Understanding  team  performance  becomes  even  more  important  for  NCW 
because  technological  advances,  globalization,  organizations,  and  global 
militaries  are  moving  toward  utilizing  more  network  structures  and  team-based 
functions  (Lipnack  &  Stamps,  1997).  The  term  “virtual  team”  is  becoming  more 
prevalent  as  teams  move  from  being  primarily  “co-located,”  where  team 
members  are  in  one  physical  location,  to  “virtual,”  where  team  members  may  be 
located  anywhere  on  the  globe,  separated  by  space  and  time. 

For  definition  purposes,  virtual  teams  will  be  considered  groups  of 

individuals  who  interact  through  various  communication  technologies  to 

accomplish  common  goals.  Virtual  learning  teams  are  being  used  in  education 

and  corporate  training  programs  in  an  attempt  to  enhance  collaboration  and 

cooperative  learning  experiences.  Even  though  various  studies  of  groups  using 

computer-mediated  communication  have  contributed  to  an  increased 
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understanding  of  both  face-to-face  and  virtual  teams,  the  results  are  inconclusive 
(McGrath  &  Hollingshead,  1994).  Most  studies  comparing  face-to-face  groups 
and  groups  using  communication  technology  suggest  that  face-to-face  teams  are 
more  effective.  McGrath  and  Hollingshead  (1994)  examined  fifty  studies  on 
computer-assisted  group  performance  and  found  that  computer-mediated  groups 
tend  to  have  fewer  interactions  and  less  information  exchange  among  members 
than  face-to-face  groups.  Virtual  team  members  can  exchange  verbal  information 
as  efficiently  as  a  face-to-face  team,  but  they  lack  the  ability  to  convey  non-verbal 
cues  as  easily  as  individuals  that  are  co-located.  This  lack  of  non-verbal 
communication  can  contribute  to  increased  misunderstanding  among  members 
(Warkentin,  Sayeed,  &  Hightower,  1997).  Research  also  indicates  that  face-to- 
face  teams  have  better  internal  leadership  and  coordination  than  virtual  teams 
(Burke  &  Chidambaram, 1994;  Eveland  &  Bikon,  1989). 

Theoretical  development  and  empirical  research  are  needed  to  better 
understand  and  respond  to  the  challenges  that  virtual  teams  face  (Furst, 
Blackburn,  &  Rosen,  1999).  Although  there  have  been  several  related  studies  in 
this  area,  few  research  efforts  have  focused  on  rapid-forming,  ad-hoc  successful 
teams.  In  the  case  of  real  world  examples,  which  are  often  investigated  after  the 
fact,  these  temporary,  newly-formed  team  members  have  a  limited  history  of 
working  together  and  have  few  prospects  of  working  together  in  the  future. 
Therefore,  the  question  exists:  how  is  it  that  they  succeed? 


D.  DISTRIBUTED  SOCIAL  NETWORKS 

Many  organizations  are  similar  to  networked  communities  in  that  they 

have  multiple  sets  of  work  team  members  (including  multiple  supervisors), 

geographically  seperated  relationships,  and  teams  of  co-workers  shifting  by  the 

day  and  week  as  employees  get  involved  in  multiple  projects.  The  situation  is 

different  from  that  dealt  with  by  traditional  organizational  theory,  which  defines 

organizations  as  densely  knit  workgroups  neatly  structured  in  bureaucratic, 

hierarchical  organizational  trees  (Wellman,  1997).  How  do  people  work  together 

in  large,  sprawling,  networked  organizations  where  they  are  members  of  multiple, 
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transitory,  physically  dispersed  teams  simultaneously?  In  particular,  how  do 
people  in  such  organizations  obtain  knowledge  from  others  when  they  do  not 
know  whom  to  ask? 

These  questions  are  of  immediate  practical  importance  for  NCW.  The  key 
to  making  NCW  effective  is  not  only  in  connecting  geographically  dispersed 
forces,  retrieving  information  from  sensors,  human  intelligent  agents  and  other 
information  sources,  but  also  analyzing  different  information,  generating  useful 
knowledge  and  then  distributing  it  to  the  right  person  at  the  right  time  in  the  right 
format.  This  enormous  task  explains  why  computer-supported  solutions  are 
being  developed  to  work  through  trusted  interpersonal  relationships  to  identify, 
locate,  and  receive  information  within  and  between  communities  and 
organizations  (Wellman,  1997).  It  is  not  surprising  that  research  in  knowledge 
management  (KM)  has  been  driven  by  computer  scientists  and  communication 
scientists  interested  in  building  tools  for  harvesting  knowledge  and  then 
managing  the  growing  database.  The  critical  issue  of  tracking  who  knows  what 
information  at  any  given  time  is  a  more  complex  task  in  networked  organizations 
(Cross  &  Borgatti,  2000).  Traditionally  in  an  office  setting,  an  individual  will  first 
examine  documentation  or  other  help  sources  from  their  desks  before  wandering 
out  into  a  hallway  in  search  of  friendly  colleagues.  The  problem  becomes  acute, 
however,  in  distributed  communities  (Ackerman  &  McDonald,  1996). 


E.  SITUATIONAL  AWARENESS 

Situational  awareness  (SA)  is  a  concept  that  grew  from  lessons  learned  in 
the  battlefield.  SA  has  been  defined  as  “adaptive,  externally  directed 
consciousness”  (Dzierzanowski,  1999).  The  United  States  Army  has  defined  it 
as  the  ability  to  maintain  a  constant,  clear  mental  picture  of  the  tactical  situation 
(Dzierzanowski,  1999).  This  mental  picture  includes  an  understanding  of  the 
terrain  and  the  relationship  between  friendly  and  enemy  forces 
(Dzierzanowski,  1 999). 
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SA  as  defined  by  Endsley  as:  “...the  perception  of  the  elements  in  the 
environment  within  a  volume  of  time  and  space,  the  comprehension  of  their 
meaning,  and  the  projection  of  their  status  in  the  near  future”  (Endsley,  1995; 
Wickens  et  al.,  2001). 

The  definition  by  Endsley,  as  applied  to  an  object  in  the  environment  tries 
to  answer  these  questions:  what  is  it,  what  is  it  doing,  why  is  it  doing  it,  and  what 
will  it  do  next.  However,  this  definition  misses  an  important  element  of  SA  which 
is  the  context  within  one’s  own  situation  and  not  just  an  awareness  of  external 
objects.  SA  should  also  include  an  acknowledgement  of  one’s  own 
psychological  state  (Omedei,  Wearing,  McLennan  &  Hansen,  2001).  Not  all 
objects  in  the  environment  need  to  be  monitored  all  the  time.  At  any  given  time 
some  elements  will  be  less  important  than  others,  but  Endlsey  suggests  that 
some  level  of  awareness  is  required  of  each  element,  to  monitor  changes  in 
levels  of  importance  (Endsley,  1993). 

Another  view  of  SA  is  that  it  is  a  detailed  appreciation  of  what  is 
happening  now  and  what  is  likely  to  happen  in  the  immediate  future.  This 
statement  can  be  compared  with  commander’s  intent  which  focuses  on  the 
overall  end  state  to  be  eventually  achieved  from  an  understood  start  state. 

According  to  Endsley,  an  operators’  SA  exists  in  one  of  the  following  three 
levels  of  complexity  (1 995,  2001 ): 

Level  1  consists  of  perception  of  elements  in  the  environment. 

Level  2  involves  comprehension  of  the  current  situation. 

Level  3  incorporates  the  prediction  of  the  future  actions  of  data  elements. 

Considerable  effort  has  been  invested  in  devising  methods  of  quantifying 
SA,  measuring  SA  and  training  to  improve  the  ability  to  acquire  SA  (Gawron, 
2000;  Nofi,  2000;  Endsley,  2001;  Endsley  &  Robertson,  2000).  Methods  of 
measuring  SA  range  from  the  highly  intrusive  to  the  deliberately  unobtrusive.  In 
the  former,  the  activity  is  suddenly  stopped  and  the  immediate  SA  of  participants 
is  captured  in  some  way  usually  a  series  of  questions  to  be  answered  for  later 
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comparisons  with  ground  truth  at  that  time.  One  issue  with  this  method  is  that  it 
interrupts  the  activity,  possibly  degrading  the  level  of  immersion  the  participant 
has  in  the  exercise/experiment.  Unobtrusive  measures  typically  require 
participants  to  self  rate  their  SA  retrospectively.  This  post  hoc  reporting 
procedure  relies  on  participants’  memories  of  events  that  may  have  occurred 
much  earlier  in  time.  SA  at  any  point  in  time  is  unlikely  to  be  fixed  in  long  term 
memory  because  SA  is  a  process  that  ebbs  and  flows  with  ground  truth.  A 
retrospective  query  to  asses  SA  may  provide  an  average  SA  score  over  the 
entire  scenario  or  a  specific  score  for  a  single  point  in  time. 


F.  SITUATED  COGNITION 

NCW  theory  makes  a  fundamental  assumption  that  improved  information 
infrastructures  will  facilitate  military  decision-making  and,  therefore,  military 
effectiveness.  Previous  work  linking  NCW  applications  to  military  effectiveness 
efforts  have  had  difficulties  in  modeling  the  decision-making  aspects  of  the 
process  (Hazen  et  al.,  2003).  Advocates  of  NCW  suggest  the  prime  advantage  of 
NCW  systems  is  in  the  generation,  communication,  and  mining  of  operational 
information. 

To  test  these  premises,  it  is  necessary  to  empirically  assess  the 
contribution  of  any  new  technology  and  other  contributions  to  the  NCW  system 
such  as  training  and  personnel  selection.  Therefore,  a  model  which  assesses 
the  effectiveness  of  an  NCW  system  must  include  a  decision-making  model. 
Models  of  information  throughput  and  human  decision-making  should  inform 
decision  makers  of  the  capability  of  weapon  systems.  These  models  should  not 
assume  that  all  information  is  accurate,  understandable,  timely,  cognitively 
matched,  and  fully  used  by  the  decision-maker.  In  fact,  previous  research 
indicates  that  it  is  not  the  quantity  or  quality  of  the  data  but  rather  the  contextual 
nature  of  the  information  that  is  important  in  determining  value  to  a  decision¬ 
maker  (Fewell  &  Hazen,  2004). 
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The  Dynamic  Model  of  Situated  Cognition  (DMSC)  provides  a  conceptual 
model  that  is  useful  for  modeling  military  decision  making  (Miller  &  Shattuck, 
2004;  Miller  &  Shattuck,  2005b).  The  schema  used  for  classifying  decisions  is 
based  on  Naturalistic  Decision  Making  (NDM)  theory  (Klein,  1997;  Shattuck  & 
Miller,  2005).  In  this  approach,  process  tracing  provides  a  robust  understanding 
of  contributions  of  both  the  operators  and  the  system  to  a  decision-maker’s 
actions  (Miller  &  Shattuck,  2004;  Miller  &  Shattuck,  2005a;  Shattuck  &  Miller, 
2004).  The  DMSC  is  composed  of  6  ovals  and  three  lenses  (see  Figure  1). 


All  data  in  the 
environment 


Projection  of 
decision  maker 

D  Miller  and  Shattuck,  2003 


Figure  1 .  Dynamic  Model  of  Situated  Cognition  (From:  Miller  &  Shattuck, 

2004). 

Oval  1  consists  of  all  data  in  the  battle  space  to  include  blue  and  red 
forces,  non-combatants,  terrain  features,  weather,  and  sensor  location.  Oval  2 
represents  the  data  collected  by  the  sensors.  Oval  2  is  smaller  than  Oval  1  due 
to  the  fact  that  no  sensor  array  will  detect  everything  in  the  environment.  Oval  3 
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represents  data  displayed  on  operator  screens.  Oval  4  is  the  operator’s 
perception  of  the  data.  Oval  5  represents  the  comprehension  while  Oval  6  is  a 
projection  of  current  data. 


The  model  also  has  3  lenses  (labeled  A,  B,  C  in  Figure  1)  which  mediates 
how  information  is  processed  by  the  decision  maker.  According  to  the  model,  the 
lenses  direct  the  attention  of  the  decision  maker  toward  certain  data,  and  in  some 
cases,  skew  what  information  is  perceived  (See  Figure  2).  Thus,  as  lenses 
change  shape,  attention  will  shift;  decisions  may  be  influenced  by  the  change  in 
these  lenses  (Miller  &  Shattuck,  2006). 


©  Miller  and  Shattuck, 


Figure  2.  Distortions  in  the  lenses  lead  to  inaccurate  perceptions, 
comprehensions,  and  projections  (From:  Miller  &  Shattuck,  2006). 


Miller  and  Shattuck  suggest  that  each  lens  has  a  purpose  which  is  listed  in 
Table  1  and  that  there  are  currently  6  classes  of  information  within  the  lenses  as 
can  be  seen  in  Table  2  (Miller  &  Shattuck,  2005). 
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Table  1 .  Purpose  of  Each  Lens. 


Lens 

Proposed  Purpose 

Lens  A 

Directs  attention 

Lens  B 

Mediates  information  organization 

Lens  C 

Course  of  action 

Table  2.  Classes  of  Information  Embedded  in  Each  Lens. 


•  Individual  states  and  traits 

•  Social  factors 

•  Local  context 

•  Plan 

•  Guidelines 

•  Experience _ 


Enhancements  to  the  original  model  include  the  addition  of  quantifying 
accuracy  and  certainty,  feedback  loops  (see  Figure  3)  that  represent  the  iterative 
nature  of  decision-making,  and  to  teams  (see  Figure  4)  (Miller  &  Shattuck,  2006). 


Figure  3.  Feedback  Loops  in  the  DMSC  (From:  Miller  &  Shattuck,  2005). 
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In  the  case  of  NCW  it  is  important  to  have  a  model  that  incorporates  not  only 
each  technological  system  input,  but  also  the  various  inputs  from  each  human 
node  in  the  system.  NCW,  by  its  very  nature,  is  considered  to  be  a  collaborative 
effort  and  as  can  be  seen  in  Figure  4,  the  DMSC  is  capable  of  representing  these 
team  interactions. 


Figure  4.  DMSC  Applied  to  Teams  (From:  Miller  &  Shattuck,  2005). 

The  merit  of  the  DMSC  model  has  been  recognized  by  multiple 
organizations.  However,  to  fully  test  the  model  and  explore  its  use  as  a  method 
to  evaluate  complex  systems,  there  needs  to  be  a  reliable  and  accurate  method 
for  determining  the  contents  of  each  of  the  six  Ovals  of  the  DMSC.  One 
candidate  tool  for  accomplishing  this  is  a  situated  cognition  measurement  tool 
called  the  Geographical  Recall  and  Analysis  of  Data  in  the  Environment 
(GRADE).  GRADE  was  conceived  as  a  measure  to  illuminate  what  participants’ 
were  attending  to  during  a  Future  Combat  Systems  experiment  (Miller  & 
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Shattuck,  2006).  Currently  GRADE  provides  only  qualitative  data  to  the 
researcher,  but  that  data  provides  situated  cognition  over  time  of  what  each 
participant  knew  of  the  battlespace  and  when  they  knew  it.  GRADE  has 
continued  to  demonstrate  efficacy  in  populating  Oval  4  in  multiple  field  studies 
(Miller  &  Shattuck,  2003;  Miller  &  Shattuck,  2005;  Miller  &  Shattuck  2006). 

This  thesis  describes  using  GRADE  as  method  and  process  by  which  the 
DMSC  model  can  be  validated  and  applied  in  the  field  setting  for  NCW  research. 
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III.  METHODS 


A.  OVERVIEW 

As  previously  stated,  there  is  not  a  well-defined  methodology  for 
evaluating  the  human  and  technological  components  of  a  network  system.  This 
shortcoming  in  NCW  makes  it  difficult  to  determine  whether  hardware,  software, 
or  changes  to  organizational  structures  or  procedures  really  improve  warfighting 
capabilities.  This  thesis  uses  the  DMSC  as  the  theoretical  framework  for  NCW 
and,  in  addition,  proposes  GRADE  an  evaluation  strategy  that  is  based  on  that 
model.  GRADE  examines  the  contents  of  each  of  the  six  ovals  in  the  model  to 
determine  who  knew  what  and  when  they  knew  it.  This  methodology  also 
nominates  appropriate  metrics  and  discusses  techniques  for  analyzing  data  in 
both  laboratory  and  field  settings. 

While  GRADE  has  been  used  in  several  laboratory  and  field  experiments 
to  assess  command  and  control  activities  (Miller  &  Shattuck,  2007),  the  tool  itself 
has  not  been  assessed  to  determine  its  efficacy  in  such  data  collection  venues. 
GRADE  provides  a  method  to  collect  a  participant’s  situated  cognition  and  then 
use  that  data  to  populate  Ovals  4  and  5  of  the  DMSC.  With  Ovals  one  through 
five  populated  it  is  then  possible  to  trace  how  data  and  information  propagate 
through  the  NCW  system  over  time. 


B.  VENUE 

The  Naval  Postgraduate  School  conducts  field  exercises  quarterly  in  the 
Tactical  Network  Topology  (TNT)  program.  This  series  of  exercises  provides  an 
opportunity  to  collect  NCW  data  in  a  field  environment.  Since  the  program  is 
receptive  to  input  from  investigators  in  the  design  and  development  of  scenarios, 
it  is  possible  to  influence  data  collection  activities.  The  Naval  Postgraduate 
School  (NPS)  TNT  Program  began  approximately  four  years  ago  with  the 
purpose  of  providing  the  opportunity  for  students  and  faculty  to  evaluate  some  of 
the  latest  technologies  in  an  operational  environment  and,  when  appropriate,  to 
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rapidly  transition  these  technologies  to  the  warfighter.  The  program  relies  heavily 
on  the  operational  knowledge  of  the  NPS  faculty  and  student  body  as  well  as  a 
very  close  working  relationship  with  United  States  Special  Operations  Command 
(USSOCOM).  Congressional  funding  from  the  Center  for  Defense  Technology 
and  Education  for  the  Military  Services  (CDTEMS)  together  with  funding  from 
USSOCOM  and  the  Office  of  Force  Transformation  (OFT)  has  permitted  one  to 
two  week  long  quarterly  field  exercises  to  be  conducted,  using  laboratories  on 
the  NPS  campus,  the  NPS  Beach  Laboratory  and  the  Center  for  Interdisciplinary 
Remotely-Piloted  Aircraft  Studies  (CIRPAS)  (Marina  Airport),  Monterey  Bay,  the 
Military  Operations  on  Urban  Terrain  (MOUT)  facility  at  Fort  Ord,  U.S.  Coast 
Guard  (Oakland)  facilities,  and  an  NPS  CIRPAS  UAV  test  facility  at  the  California 
Army  National  Guard,  Camp  Roberts,  CA. 

The  field  exercises  began  with  a  series  of  Surveillance  and  Target 
Acquisition  Network  (STAN)  exercises  which  have  transitioned  to  the  Tactical 
Network  Topology  (TNT)  Program  which  focuses  on  providing  the  soldier  on  the 
ground  the  ability  to  push  or  pull  video,  data,  text  messages,  and  use  voice  over 
IP  (VOIP)  to  communicate  with  other  soldiers  and  with  other  ISR  assets  in  both 
rural  and  urban  environments.  In  addition  the  network  provides  screens  that 
display  ‘situational  awareness’  and  blue  force  tracking.  It  utilizes  air  and  ground 
based  wireless  networks,  satellites,  unmanned  vehicles  (UAVs,  UGVs,  AUVs, 
airships,  tethered  balloons),  unattended  ground  sensors,  handheld  PDAs,  and 
laptops  to  enhance  situational  awareness  and  to  enhance  our  ability  to  find,  fix, 
and  identify  enemy  personnel  and  equipment. 

TNT  07-03  was  conducted  11  through  18  May  2007  at  Camp  Roberts, 
California.  This  field  exercise  consisted  of  the  following  scenario  activities: 
Perimeter  Security,  Border  Surveillance,  Area  Search,  and  Identify/Monitor 
Area/Activity.  The  data  for  this  research  effort  were  collected  on  the  morning  of 
17  May  2007  during  a  baseline  scenario. 

The  assigned  mission  for  the  five  members  of  the  Blue  Forces  was  to 
monitor  the  borders  for  trespassers.  In  the  event  a  trespasser  was  spotted,  they 

were  to  conduct  surveillance  and  monitor  their  activity. 
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The  assigned  mission  for  the  three  members  of  the  Red  Force  was  to 
cross  the  borders  at  different  locations.  Red  1  was  to  drop  off  the  material  for  an 
IED  at  a  certain  location.  Later  Red  2  would  stop  by  the  predetermined  location 
and  assemble  the  IED.  The  last  part  of  the  mission  was  to  have  Red  3  set  up  the 
IED  at  that  location  for  use  and  simulate  detonating  the  device. 

1.  Participants 

Data  were  collected  on  participants  in  the  following  positions  during  the  17 
May  2007  morning  scenario.  Participants  in  the  following  positions  completed 
the  GRADE  measures  and  wore  digital  audio  recorders. 

•  Tactical  Operations  Center  (TOC)  Commander 

•  Air  Boss 

•  Raven  3  (UAV) 

•  Buster  (UAV) 

•  SORSE  (five  individuals) 

•  Red  force  commander 

•  Red  vehicle  1 

•  Red  vehicle  2 

•  Red  vehicle  3 

Although  Raven  3  participated  in  the  exercise,  the  GPS  location  data  on 
this  UAV  were  never  captured  due  to  technical  difficulties  with  Cursor  on  Target 
software  making  it  impossible  to  establish  ground  truth  for  this  vehicle.  The 
Raven  3  GRADE  data  were  compared  to  the  other  GRADE  data  to  determine  if 
participants  correctly  populated  the  measures. 

Special  Operations  Research,  development,  test  &  evaluation  Support 
Element  (SORSE)  is  the  Blue  force  ground  troops  and  was  composed  of  five 
individuals.  Three  of  them  rode  on  ATVs  while  the  other  two  drove  in  SUV’s. 
Redl,  Red2,  and  Red3  comprised  the  red  force  ground  troops.  All  three  drove 
their  own  civilian  vehicles. 
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2.  Apparatus  and  Instrumentation 

As  discussed  previously  the  DMSC  is  composed  of  6  ovals.  The  first 
three  ovals  are  comprised  of  the  technological  system’s  contribution  and  the  last 
three  ovals  are  the  human  contribution  to  the  total  system.  It  follows  that 
measurements  should  follow  this  division.  Table  3  indicates  the  measures  that 
could  be  used  to  populate  these  ovals.  In  the  case  of  the  current  research 
project,  data  were  captured  primarily  for  Ovals  1 ,  2,  3,  4  and  5  of  the  DMSC  for 
the  TNT  07-03  field  experiments. 


Table  3.  Measures  used  for  populating  DMSC. 


Oval 

TNT  Field  Experiment 

1 

GPS  network 

Network  data  storage 

2 

Sensor  network 

Network  data  storage 

Sensor  capabilities 

3 

Screen  captures  for  individual 
players 

4,  5,6 

GRADE 

Digital  voice  recorders 

Acetate  overlays  (Oval  6) 

To  maintain  a  consistent  temporal  reference,  both  the  GRADE  measures 
and  digital  voice  recorders  were  synchronized  to  the  GPS  network  time.  To 
capture  data  for  Oval  1,  ground  truth  data  were  collected  from  each  vehicle’s 
GPS  tracking  system  and  stored  by  the  TNT  network.  Commercial  GPS  units 
were  used  by  the  ground  forces;  UAV  position  was  captured  by  software 
broadcast  of  GPS  position.  In  the  case  of  the  ground  troops,  the  GPS  units  were 
linked  with  laptop  computers  that  had  wireless  connectivity  with  the  TNT  network. 
GPS  data  were  sampled  at  a  rate  of  once  per  second  for  the  duration  of  the 
scenario.  Figures  5  and  6  provide  examples  of  GPS  data  collection  and  storage. 
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Figure  5.  Tech  Data. 


Figure  6.  Tech  Data  +  time. 
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Oval  2  data  were  captured  first  by  understanding  a  sensors’  capability  and 
secondly  by  providing  data  on  where  the  sensor  was  looking.  In  the  case  of  the 
former,  a  full  understanding  of  what  a  sensor  could  see  or  hear  is  taken  into 
consideration  for  correctly  identifying  hits  and  misses  of  participants.  For 
example,  an  audio  sensor  is  not  going  to  provide  a  picture  enabling  participants 
to  visually  ID  a  vehicle;  similarly,  a  small  UAV  flying  at  10,000  feet  is  not  going  to 
have  the  resolution  required  to  allow  identification  of  moving  ground  targets.  In 
the  latter  case,  it  is  not  enough  to  know  where  a  sensor  is,  but  it  is  also  essential 
to  know  where  it  is  looking.  The  Scan  Eagle  UAV  was  capable  of  providing  this 
capability  by  providing  its  own  location  and  the  location  where  the  camera  was 
looking.  This  was  achieved  by  software  that  could  calculate  the  location  the 
camera  was  looking  by  calculating  camera  position  versus  UAV  location.  With  a 
field  of  view  calculation  researchers  could  theoretically  determine  the  exact  area 
viewed  by  the  sensor  throughout  an  exercise.  Unfortunately,  the  Scan  Eagle 
UAV  did  not  participate  in  the  17  May  AM  scenario. 

Oval  3  was  populated  by  collecting  screen  captures  of  what  was  on 
participants’  displays  at  fifteen  minute  intervals  (see  Figures  5  and  6). 

Oval  4  and  5  data  was  captured  by  a  combination  of  both  the  situated 
cognition  measure  GRADE  and  digital  voice  recorders.  GRADE  was  collected  in 
this  case  using  a  map  of  the  area  of  operation  for  the  TNT  07-03  experiments 
(see  Figure  7).  Multiple  blank  GRADES,  a  clipboard,  and  a  blue  and  red  Sharpie 
fine  point  permanent  marker  were  provided  to  each  participant. 
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Figure  7.  GRADE  example. 

Nine  Olympus  WS-300M  digital  voice  recorders  were  used  to  collect  audio 
data  from  the  participants.  To  ensure  the  maximum  understanding  of  voice  data, 
omnidirectional  microphones  were  used  in  conjunction  with  the  digital  voice 
recorders. 

3.  Procedure 

The  procedure  section  is  organized  to  indicate  how  data  was  collected  for 
each  Oval  of  the  DMSC  during  the  TNT  07-03  field  experiment.  This  section  will 
start  with  Oval  one  and  conclude  with  Oval  five. 

a.  Oval  1 

Careful  advance  collaboration  with  TNT  exercise  planners  enabled 
an  understanding  of  what  sources  of  data  were  available  to  be  collected  and  then 
used  to  populate  the  first  three  Ovals.  For  Oval  1  all  UAVs  and  ground  vehicles 
were  equipped  with  GPS  units  that  were  able  to  broadcast  to  the  TNT  network. 
Location  data  was  sampled  and  saved  once  per  second. 

b.  Oval  2 

For  Oval  2  all  data  provided  by  technological  sensors  were  time 
stamped  as  the  information  was  passed  into  the  TNT  network. 

c.  Oval  3 

For  Oval  3  a  screen  capture  of  what  was  presented  on  the  displays 

was  captured  every  15  minutes.  In  addition,  in  the  Tactical  Operations  Center  a 
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time-stamped  digital  log  was  kept  of  what  UAV  video  feed  observers  noted. 
Finally,  the  TOC  CDM  and  Air  Boss  wore  digital  voice  recorders  providing  further 
information  on  what  information  was  available  on  the  displays. 

d.  Oval  4  and  5 

The  decision  makers  listed  in  the  Participants  section  (above)  were 
trained  in  the  GRADE  protocol.  The  GRADE  was  administered  every  twenty 
minutes  during  the  exercise.  The  Air  Boss  would  announce  over  the  radio 
network  time  to  complete  a  GRADE  to  the  Blue  forces  while  Red  Commander 
would  make  the  same  announcement  to  Red  forces.  Participants  were  instructed 
to  provide  their  understanding  of  the  battle  space  on  the  GRADE  maps  (see 
appendix  A).  The  GRADE  instructions  were  as  follows: 

Please  complete  the  GRADE  every  20  minutes  after  STARTEX 
until  FINEX  according  to  the  following  instructions. 

1 .  Turn  away  from  your  information  display,  monitor  etc. 

2.  Record  your  duty  positions  (e.g.,  Air  Boss,  Redl,  etc.)  and  the 
current  time  in  the  upper  left  portion  of  the  map. 

3.  Use  the  blue  marker  to  identify  where  you  believe  the  blue 
forces  are  located.  Draw  a  circle  around  each  blue  entity  to 
indicated  your  confidence/certainty.  A  small  circle  indicated 
high  confidence/certainty.  A  large  circle  indicates  low 
confidence/certainty. 

4.  Use  the  red  marker  to  identify  where  you  believe  the  red 
forces  are  located.  Draw  a  circle  around  each  red  entity  to 
indicate  your  confidence/certainty.  A  small  circle  indicates 
high  confidence/certainty.  A  large  circle  indicates  low 
confidence/certainty. 

The  top  left  corner  of  the  GRADE  map  indicated  the  TNT 
experiment  and  date  of  data  collection.  Blanks  were  provided  for  participant 
position  information  and  time  the  GRADE  was  conducted. 
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4.  Data  Analysis  Plan 

Prior  to  collecting  the  data  at  Camp  Roberts,  a  plan  for  analyzing  the 
GRADES  was  developed.  The  plan  included  converting  latitude/longitude  data 
provided  by  GPS  devices  to  military  grid  coordinates.  Then,  a  master  location 
spreadsheet  would  be  created  that  listed  the  military  grid  coordinates  for  all 
relevant  entities  in  the  battlespace  for  each  time  the  GRADE  was  administered. 
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Figure  8.  Location  data  spreadsheet. 


Military  grid  coordinates  for  each  participant’s  entries  on  the  GRADE  sheets 
would  also  be  determined.  The  participants’  marks  for  each  entity  would  be 
compared  with  the  actual  (ground  truth)  location  and  the  distance  between  the 
actual  (ground  truth)  and  the  recorded  (actual)  would  be  calculated  (see  Figure  8 
for  an  example). 

With  the  DMSC  populated  with  data  it  is  now  possible  to  use  the  GRADE 
data  (Ovals  4  and  5)  to  look  back  through  Ovals  1,  2,  and  3  to  determine  the 
errors,  if  there  were  any,  in  a  participants’  perceived  ground  truth  of  the 


Time 

Easting  1  Military  Grid  1  Northing 

Military  Grid 

ID 

Notes 

Ground  Truth  East 

Ground  Truth  North 

9:50 

100 

6000 

B 

B 

98345 

58897 

9:50 

10 

5740 1 

9:50 

9920 

58601 

9:50 

450 

61901 

9:50 

150 

61401 

9:50 

9:50 

280 

6130 

B 

9:50 

140 

6080 

Buster 

190 

61101 

9:50 

120 

6110 

White  SUV 

9790 

5920 1 

9:50 

130 

6090 1 

9:50 

140 

61201 

10:14 

100 

6000 

B 

9830 

5890 1 

10:14 

10 

5740 1 

10:14 

9920 

5860 1 

10:14 

450 

61901 

10:14 

150 

61 40 1 

10:14 

10:14 

10:14 

170 

621  ol 

10:14 

9950 

6080 

Black  SUV 

810 

5840 1 

10:14 

130 

6090 

Black  SUV 

9990 

61 90 1 

10:14 

140 

6180 

White 

130 

6080 1 

10:33 

9880 

5990 

stationary  reserves 

9830 

~  5890 1 

10:33 

o 

57401 

10:33 

9920 

5860 1 

10:33 

450 

61 90 1 

10:33 

160 

61 40 1 

10:33 

10:33 

10:33 

120 

6240 1 

10:33 

9860 

6040 

Black  SUV 

310 

6350 1 

10:33 

9860 

6040 

White  SUV 

9890 

6060 1 

10:30 

130 

61 90 1 

10:50 

9840 

58901 

10:50 

0 

5740 1 

10:50 

9920 

5860 1 

10:50 

450 

61 90 1 

10:50 

150 

61 40 1 

10:50 

180 

551  Ol 

10:50 

190 

5520 1 
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battlespace.  For  example,  a  researcher  may  notice  a  point  where  a  participant 
incorrectly  identified  a  vehicle.  The  data  trace  would  possibly  provide  this 
information:  a  Red  force  unit  crosses  a  border  (Oval  1),  but  is  not  yet  detected.  A 
UAV  (sensor)  flies  by  the  enemy  capturing  an  image  (Oval  2).  The  video  feed 
flows  through  the  network  and  is  displayed  on  an  observers  screen  (Oval  3).  The 
participant  notices  the  vehicle  (Oval  4)  and  then  incorrectly  labels  it  a  Blue  force 
vehicle  (Oval  5).  Now  the  researcher  can  try  to  determine  where  the  error  was 
introduced.  Was  it  because  the  display  did  not  provide  enough  resolution  (Oval 
3)  or  due  to  the  lens  between  Ovals  4  and  5  indicating  to  the  participant  only  Blue 
forces  are  in  that  area. 


Figure  9  illustrates  the  GRADE  measures  ability  to  capture  a  participant’s 
situated  cognition  (Ovals  4  and  5)  over  time.  These  figures  include  the  ground 
truth  overlays. 


SORSE  GRADE  3  SORSE  GRADE  4+20  SORSE  GRADE  5+20 

Figure  9.  GRADE  data  and  ground  truth  from  SORSE  at  approximate  20 

minute  intervals. 
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IV.  RESULTS 


Due  to  the  high  rate  of  missing  data,  quantitative  statistical  analysis  could 
not  be  completed  for  this  thesis.  Instead,  a  descriptive  analysis  of  Participants’ 
performance  on  GRADE,  ground  truth  vs.  perceived  truth,  and  a  qualitative 
analysis  of  the  digital  voice  recordings  is  provided. 

A.  GRADE  DATA 

The  data  collected  from  GRADE  immediately  indicated  to  the  researchers 
that  there  are  issues  in  how  the  participants  populated  the  GRADE  over  the 
course  of  the  scenario.  As  table  XX  indicates,  nine  participants  completed  57 
GRADE  measures  over  the  course  of  the  two  hour  experiment.  Out  of  the  57 
GRADES  generated,  not  a  single  GRADE  was  completed  correctly  indicating  that 
participants  did  not  follow  instructions.  To  salvage  the  data  collection,  it  was 
determined  that  if  at  least  one  Red  or  Blue  force  was  clearly  labeled,  that 
GRADE  would  be  considered  usable.  With  this  new  criterion,  21  GRADE 
measures  provided  useful  data  indicating  a  37%  ratio  for  completed  GRADES 
(see  Table  4). 
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Table  4.  Total  GRADES  vs  usable  GRADES. 


Position 

GRADES  Completed 

Usable  GRADES 

Air  Boss 

7 

0 

Buster  (UAV) 

7 

2 

Raven  (UAV) 

6 

0 

Red  Queen 

6 

0 

Red  1 

6 

5 

Red  2 

6 

6 

Red  3 

6 

4 

SORSE 

6 

4 

TOC  CC 

7 

0 

TOTAL 

57 

21 

Further  analysis  of  the  21  usable  GRADES  indicates  only  44  usable  data 
points  out  of  a  possible  282  data  points  resulting  in  16%  usable  data  for  this 
research  effort.  This  result  is  artificially  low  due  to  the  assumption  that 
participants  would  know  where  all  players  were  at  all  times.  A  more  accurate 
measure  of  GRADE  performance  is  achieved  by  contrasting  the  44  usable  data 
points  against  the  131  actual  data  points  found  on  the  GRADE  maps.  In  this 
case  the  GRADE  measure  provided  34%  usable  data  return. 

As  Table  5  indicates,  there  were  a  possible  513  total  detections  during  the 
course  of  this  scenario.  Participants  only  indicated  262  actual  detections  yielding 
a  51%  success  rate  of  detecting  either  red  or  blue  forces.  Both  Blue  (105)  and 
Red  (50)  forces  did  better  at  detecting  themselves  than  at  detecting  the  opposing 
force. 

Blue  force  had  a  total  of  167  detections  indicated  on  the  GRADE 


measures.  62  of  those  detections  were  of  Red  forces  while  the  remaining  105 
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were  of  friendlies  indicating  a  37%  rate  of  detecting  enemy  forces.  Red  force 
had  fewer  players  in  the  scenario,  so  it  is  not  surprising  that  their  total  detection 
of  95  is  lower  than  the  Blue  force.  Red  force  had  45  Blue  force  detections 
indicating  a  47%  rate  of  enemy  force  detections. 


Table  5.  Detections. 


Detections 

Blue 

Red 

Total 

Possible 

342 

171 

513 

Actual 

150 

112 

262 

By  Blue 

105 

62 

167 

By  Red 

45 

50 

95 

Table  6  indicates  a  different  result  for  the  TOC  Commander  who  has  the 
potential  capability  of  having  positions  for  all  6  Blue  force  assets  and  3  Red  force 
assets.  The  same  measurements  already  described  were  used  to  populate  this 
table. 


Table  6.  Qualitative  examination  of  TOC  CDR  GRADE  data. 


Ground  force 

Know  Self/friendly 

Enemy 

TOCC  CC  (Blue) 

6 

3 

At  first  glance  the  table  would  lead  one  to  believe  that  the  TOC 
Commander  had  an  accurate  representation  of  the  battle  space.  This  is  an 
artifact  due  to  averaging  the  Blue  and  Red  force  marks.  The  raw  data  indicate  a 
different  story  (see  Table  7).  In  fact,  there  were  GRADES  that  indicated  up  to  8 
Blue  force  locations  (there  were  only  6  Blue  force  entities)  and  up  to  4  Red  force 
locations  (there  were  only  3  Red  force  players). 
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Table  7.  TOC  Commander  Raw  Data. 


Blue  Red  Total 

8  0  8 

5  3  8 

6  2  8 

6  2  8 

5  4  9 

5  4  9 

5  3  8 


B.  GROUND  TRUTH  VS.  PERCEIVED  TRUTH 

Due  to  the  numerous  incorrectly  completed  GRADES  and  the  different 
times  GRADE  measures  were  completed,  it  was  impossible  to  create  a  common 
picture  of  the  battlespace  at  any  point  in  time.  The  next  best  solution  was  to 
determine  if  there  were  participants  who  identified  each  other  at  close  temporal 
points.  This  criterion  change  yielded  three  participants  who  had  completed 
GRADES  within  seven  minutes  of  each  other  (see  Table  8).  This  data  indicates 
that  within  a  seven  minute  period,  the  Blue  force  knew  the  exact  location  of  Red 
2  and  Red  3.  This  battlespace  knowledge  was  better  than  Red  1  (Red  2  error  = 
100m;  Red  3  error  =  200m)  or  Red  2  (Red  2  error  =  200m;  Red  3  error  =  200m) 
knowledge  of  the  battlespace. 


Table  8.  Raw  data  near  temporal  point  one. 


Participant 

Time 

GRADE  # 

Vehicle 

Error  in  meters 

SORSE 

9:50 

1 

R2 

0 

9:50 

1 

R3 

0 

Redl 

9:44 

2 

R2 

100 

9:44 

2 

R3 

200 

Red2 

9:43 

2 

R2 

200 

9:43 

2 

R3 

200 

One  hour  later  as  seen  in  table  9,  the  same  three  participants  completed 
GRADES  at  nearly  the  same  time.  This  data  set  spans  a  six  minute  period  in 
which  the  Blue  force  lost  Red  2,  detected  Red  1  (Red  1  error  =  700m)  and  had 


32 


more  error  in  Red  3;  Red  3  error  =  300m).  Both  Red  1  (Red  2  error  =  500m;  Red 
3  error  =  1300m)  and  Red  2  (Red  2  error  =  300m;  Red  3  error  =  800m)  knew  of 
their  locations. 


Table  9.  Raw  data  near  temporal  point  two. 


Participant 

Time 

GRADE  # 

Vehicle 

Error  in  meters 

SORSE 

10:50 

4 

R1 

700 

10:50 

4 

R3 

300 

Red  1 

10:44 

5 

R2 

500 

10:44 

5 

R3 

1300 

Red  2 

10:50 

5 

R2 

300 

10:50 

5 

R3 

800 

C.  AUDIO  DATA 

The  audio  recordings  from  the  following  four  participants  were  transcribed: 
TOC  Commander,  SORSE,  Red  1,  and  Red  2. 

An  overall  analysis  of  the  data  from  Blue  forces  indicates  that  Blue 
participants  primarily  reported  on  the  detections  and  locations  of  Red  forces  and 
Red  activities.  There  were  four  instances  of  military  grid  coordinates  on  Red 
forces  being  passed  over  the  Blue  radio  network  and  many  referrals  to  land 
marks  near  the  Red  forces  (e.g.,  under  tree  or  near  ranch  house).  One 
interesting  question  from  the  Buster  UAV  ground  control  unit  to  SORSE  ground 
forces  was,  “Have  you  reported  any  of  these  things  yet?  I’ve  just  been  sending 
them  [pictures].” 

TOC  Commander  audio  logs  indicated  that  he  provided  very  little  input  to 
SORSE.  In  fact  there  is  only  one  instance  in  the  two  hour  scenario  that  he 
provides  information  to  SORSE  about  a  vehicle  coming  down  a  road. 

SORSE  audio  logs  indicated  some  confusion  coming  from  the  Buster 
ground  control  unit  about  what  pictures  to  take.  There  is  one  instance  at  10:10 
am  of  Blue  3  passing  an  incorrect  report  on  a  Red  vehicle  location  and  its 
activities. 
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An  overall  analysis  of  the  Red  forces  voice  recordings  indicates  that  Red 
forces  primarily  reported  on  their  own  locations  and  activities.  Red  1  audio  logs 
indicated  that  he  could  hear  the  Blue  forces  radio  net.  In  fact,  when  Blue  forces 
would  report  detecting  a  Red  player,  Red  1  would  report  in  to  Red  commander 
that  Blue  forces  were  correct.  Red  1  had  the  least  amount  of  data  out  of  these 
four  analyses. 

Red  2  audio  logs  indicated  occasional  UAV  sightings,  but  mainly  consist  of 
comments  on  Red  force  locations  and  activities.  Red  2  was  very  concerned  with 
STARTEX  and  FINEX  times  for  the  scenario. 
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V.  DISCUSSION 


Some  members  of  the  Department  of  Defense  believe  the  NCW  assertion 
that  technology  will  provide  a  multiplicative  effect  to  US  military  forces  resulting  in 
improved  capability  and  mission  effectiveness.  However,  to  date  there  are  no 
empirical  findings  that  support  or  refute  the  claims  put  forth  by  the  architects  of 
NCW. 

This  thesis  proposed  an  objective  method  to  evaluate  NCW  system 
assumptions  by  objectively  assessing  system  performance  using  a  process 
tracing  approach.  Data  collection  occurred  on  17  May  2007  in  the  field  at  Camp 
Roberts,  California  during  TNT  07-03  field  experiment.  Due  to  a  high  data  loss, 
statistical  analysis  for  supportable  conclusions  could  not  be  drawn  from  the 
GRADE  measure.  Thus  the  DMSC  could  not  be  validated  by  this  thesis  in  a  field 
setting. 

Following  is  an  analysis  of  what  went  well  and  what  went  awry  with  the 
overall  GRADE  administration  including  instructions  for  GRADE,  adequacy  of 
training  for  participants  filling  out  GRADE,  quality  control  issues  for  the  GRADE, 
and  finally,  a  discussion  on  the  scope  of  the  experimental  data  collected. 


A.  GRADE  DATA 

Many  factors  contributed  to  the  inability  of  the  GRADE  measure  to  provide 
usable  data.  An  after  action  review  of  the  researchers’  processes  and 
procedures  to  populate  the  DMSC  and  evaluate  the  GRADE  measure  provided 
the  following  items  as  factors  that  hampered  the  data  collection  effort. 

Oval  1  -  Ground  Truth  was  collected  by  having  all  vehicles  equipped  with 
GPS  that  broadcasted  their  locations  to  the  TNT  network.  After  an  initial  review 
of  the  GPS  data  it  was  noted  that  there  were  some  issues  with  this  technological 
solution  establishing  ground  truth.  Multiple  vehicles  had  time  periods  for  which 
no  GPS  data  were  recorded;  the  Raven  UAV  yielded  no  usable  data.  Access  to 
GPS  data  for  a  fast  moving  vehicle  (e.g.,  a  UAV  or  automobile)  is  critical  to 
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accurately  place  it  in  the  battlespace  over  the  course  of  a  scenario.  The  need  for 
frequently  updated  GPS  location  data  for  vehicles  that  are  moving  slowly  or 
standing  still  is  less  critical. 

If  the  DMSC  is  to  be  a  viable  approach  to  the  assessment  of  NCW  it  is 
important  to  consider  the  issues  and  challenges  associated  with  establishing 
ground  truth.  GPS  systems  were  the  technological  method  chosen  to  record  the 
participants’  locations  at  the  TNT  event.  Although  GPS  systems  are  fairly 
accurate,  the  data  they  yield  still  has  some  degree  of  error.  There  is  a  possibility 
of  an  error  of  up  to  100  meters  from  the  physical  GPS  location  due  to  a  variety  of 
factors  (e.g.  atmospheric  conditions,  ephemeris  error,  clock  drift,  measurement 
noise,  selective  availability,  multipath,  dilution  of  precision,  obstruction). 

Oval  2  -  Sensor  Data  were  collected  by  determining  the  platform’s 
capability  and  then  by  collecting  data  on  where  the  sensor  was  actually  pointed. 
At  the  TNT  event,  researchers  had  a  full  understanding  of  the  platform’s 
capability  prior  to  the  scenario.  In  the  later  case,  there  are  two  particular  sensor 
issues  that  need  to  be  resolved..  The  first  concerns  UAVs  and  the  second  the 
human  participants. 

For  this  TNT,  there  were  GPS  locations  for  the  Buster  UAV;  however,  the 
GPS  locations  were  not  available  for  what  the  camera  saw.  This  problem 
contributed  to  errors  in  scoring  the  data  for  detections.  For  example,  it  is 
possible  the  ground  truth  data  will  indicate  Buster  flew  over  Redl  which  may  be 
could  potentially  be  a  hit.  However,  further  analysis  may  indicate  that  Buster  was 
actually  banking  hard  to  the  right  when  it  passed  over  Redl,  putting  the  camera’s 
focus  away  from  the  target.  In  terms  of  the  DMSC  model,  ground  truth  or  data 
populating  Oval  1  never  made  it  to  Oval  2  due  to  the  sensors  inability  to  detect 
Redl.  This  missing  data  will  propagate  through  the  model  presenting  a  false 
representation  of  the  battlespace  for  Oval  3  which  results  in  the  participants’ 
inability  to  perceive  this  data  for  Oval  4  and  comprehend  it  in  Oval  5.. 

Another  problem  occurred  when  trying  to  capture  where  the  human 
sensors  (i.e.,  the  exercise  participants)  were  actually  looking.  Currently,  there  is 
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not  a  process  in  place  to  collect  these  data  during  TNT  experiments.  This 
inability  to  capture  the  actual  information  perceived  by  the  participants  leads  to 
scoring  errors  when  trying  to  determine  hits,  misses,  or  other  categories.  As 
discussed  in  Chapter  III,  it  is  not  enough  to  know  where  a  sensor  platform  is 
located,  it  is  also  essential  to  know  where  it  is  looking  in  order  to  populate  Oval  2. 
This  applies  to  both  human  and  technological  sensors. 

Oval  3  -Oval  3  was  populated  by  capturing  screen  shots  of  the  Air  Boss’s 
monitor  in  the  TOC  at  15  minute  intervals.  Fifteen  minute  data  sample  intervals 
do  not  provide  enough  fidelity  to  populate  Oval  3.  Only  five  of  six  screen 
captures  were  potentially  useful,  and  of  those,  only  two  aligned  temporally  with 
two  different  participants’  GRADE  measure  completion  times.  One  of  the  main 
reasons  screen  captures  were  not  usable  was  because  they  did  not  have  time 
stamps  on  them,  making  them  difficult  to  integrate  them  into  the  other  data. 

Oval  4  &  5  -  Oval  4  (Perception)  and  Oval  5  (Comprehension)  data  were 
collected  using  the  GRADE  and  digital  voice  recorders.  In  the  case  of  the  former, 
all  participants  made  errors  in  indicating  their  understanding  of  the  battlespace 
on  the  GRADE  maps.  On  a  few  GRADES,  only  one  color  marker  (blue  or  red) 
was  used  to  identify  both  Red  and  Blue  forces.  Often  there  were  blue  or  red 
marks  but  no  identification  with  which  to  compare  that  mark  to  ground  truth.  In 
other  instances,  unique  identifiers  were  used  but  the  researchers  could  not 
determine  what  they  represented. 

There  is  not  a  consistent  temporal  referent  across  data  samples  since 
participants  did  not  complete  the  GRADE  at  the  precise  time  they  were 
instructed.  The  morning  scenario  on  17  May  07  had  an  official  start  time  of  09:04 
with  an  expected  first  GRADE  of  09:24.  Actual  initial  GRADE  start  times  ranged 
from  09:20  through  09:40.  Within  a  participants  GRADE  measures  there  was 
variability  such  that  the  GRADES  were  not  always  taken  at  twenty  minute 
intervals.  In  fact,  there  was  enough  temporal  discrepancy  that  three  participants 
completed  one  additional  GRADE. 
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The  digital  voice  recorder  data  were  not  without  error  either.  Some  of  the 
voice  recordings  were  lost  when  some  participants  accidentally  turned  off  the 
digital  voice  recorders  or  when  participants  placed  the  recorder  down  and  walked 
away,  forgetting  that  they  were  supposed  to  keep  it  on  their  bodies.  Another 
issue  with  digital  voice  recorder  data  is  the  lack  of  speech  intelligibility  due  to 
background  noise  or  wind  blowing  over  the  microphone. 


B.  GRADE  INSTRUCTIONS 

The  inconsistencies  in  how  each  participant  filled  out  their  GRADE 
measures  indicated  that  the  instructions  were  not  clear.  One  apparent  issue 
with  the  GRADE  instructions  is  that  it  was  not  clearly  stated  that  when  marking 
the  location  of  Blue  or  Red  forces,  the  participant  must  identify  the  blue  or  red 
mark.  For  example,  if  the  participant  knew  that  the  Honda  SUV  belonged  to  Red 
1  they  should  have  placed  a  red  dot  on  the  map  and  annotated  near  it,  “Red  1 

Participants  also  did  not  follow  the  first  instruction  which  stated  that 
participants  needed  to  look  away  from  their  displays  while  completing  the 
GRADE.  Two  participants’  GRADES  closely  mirrored  ground  truth,  indicating 
they  were  using  the  GRADE  to  represent  what  was  on  their  displays  (Oval  3 
data)  rather  than  where  they  were  focusing. 


C.  GRADE  TRAINING 

Due  to  inconsistencies  in  how  participants  filled  out  their  GRADES,  there 
appears  to  be  a  lack  of  understanding  by  the  participants  of  what  is  required  to 
complete  the  forms.  These  inconsistencies  clearly  indicate  inadequacy  in 
training.  This  should  not  be  surprising  since  there  were  no  formal  processes  or 
procedures  for  the  training  of  participants  in  how  to  complete  a  GRADE.  This 
inadequacy  was  due  to  time  limitations  of  participants  and  the  lack  of  personnel 
available  to  conduct  the  training.  Again,  this  is  not  surprising  since  participants  in 
the  TNT  experiments  do  not  have  much  extra  time.  Typically,  participants  are  so 
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heavily  task  loaded  to  execute  the  scenarios  they  do  not  have  time  for  a  formal 
class  before  or  between  scenarios  to  receive  training. 

The  second  reason  is  the  lack  of  personnel  available  to  participate  in  data 
collection.  Generally  one  or  two  researchers  try  to  spend  a  few  minutes  with 
each  participant  to  offer  ‘just  in  time  training’  before  a  TNT  scenario  began.  For 
the  scenario  this  thesis  investigated,  there  was  only  one  researcher  available  to 
collect  data  from  nine  participants,  most  of  whom  were  not  collocated  with  the 
researcher. 


D.  QUALITY  CONTROL  FOR  GRADE 

There  were  no  formal  processes  or  procedures  for  quality  control  checks 
of  the  GRADES  completed  by  the  participants.  This  is  primarily  due  to  the 
assumptions  that  participants  would  receive  training  in  how  to  complete  the 
GRADE  and  that  the  GRADE  was  not  difficult  to  complete.  As  already 
mentioned,  it  was  impossible  for  a  single  researcher  to  check  on  the  nine 
participants’  GRADE  measures  each  time  they  were  completed. 


E.  RESEARCH  EFFORT  SIZE 

The  inadequacy  of  the  GRADE  data  collection  resulted  from  researchers 
trying  to  capture  too  much  data  in  the  hope  of  being  able  to  reconstruct  ground 
truth  for  the  entire  two  hour  scenario.  This  goal  overstretched  the  limited 
resources  and  personnel  available  for  these  data  collection  efforts.  All  the  issues 
discussed  up  to  this  point  can  be  traced  back  to  the  problems  that  arose  when 
trying  to  collect  too  much  data  with  too  few  assets. 

In  the  event  that  this  research  effort  had  collected  all  the  data  initially 
planned,  the  total  amount  of  data  would  have  been  enormous  and  would  have 
taken  several  months  to  process  for  analysis.  For  example,  processing  the  data 
for  Ovals  4  and  5  on  the  TOC  CDR,  SORSE,  Redl  and  Red2  required 
approximately  40  hours  of  effort  from  four  full-time  research  assistants. 
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Completing  this  analysis  of  what  went  well  and  what  went  awry  with  the 
processes  and  procedures  of  administering  the  GRADE  measure  provided 
researchers  with  a  framework  in  which  recommendations  could  be  generated 
and  a  future  course  of  action  could  be  planned  for  the  next  data  collection 
opportunity. 
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VI.  CONCLUSION 


A.  CONCLUSION 

Network  Centric  Warfare  systems  need  to  be  empirically  tested  in  order  to 
validate  developers’  claims  of  enhanced  capability  and  mission  effectiveness. 
The  Dynamic  Model  of  Situated  Cognition  was  selected  as  a  way  to  model  and 
test  NCW  systems  due  to  its  proven  merit  and  recognition  by  many  organizations 
as  a  way  to  model  human  system  performance.  The  goal  of  this  research  project 
was  to  determine  if  the  GRADE  could  be  used  as  a  data  collection  strategy  and 
process  by  which  the  DMSC  model  could  be  validated  in  a  field  setting  for  NCW 
research.  Unfortunately,  that  goal  was  not  fully  realized  with  the  TNT  07-3  data 
collection  effort.  However,  it  was  not  the  DMSC  model  or  the  GRADE  that 
contributed  to  the  poor  data  collection  results,  but  rather  inadequacies  in  the 
process  and  procedures  employed  to  populate  the  DMSC  and  GRADE.  The 
results  of  this  research  effort  in  no  way  refute  the  efficacy  of  the  GRADE  or  the 
DMSC  to  model  NCW  in  the  field.  These  results  have  brought  to  light  procedural 
errors  in  the  methods  used  on  GRADE  administration. 

Following  are  recommendations  that  will  optimize  GRADE  data  collection 
in  future  TNT  experiment  opportunities.  Then,  the  next  section  describes  the 
need  for  an  automated  or  digital  GRADE,  which  is  then  followed  by  a  section 
describing  a  different  strategy  to  analyze  certain  events  of  interest. 


B.  RECOMMENDATIONS 

GRADE  instructions.  Since  the  TNT  exercise  occurs  every  quarter  or 
so,  participants  have  had  experience  in  completing  GRADES  in  past  TNT 
experiments.  Participant  errors  indicate  the  instructions  have  some  ambiguity 
and  allow  for  interpretation  of  intent.  Following  are  suggested  instructions  to 
improve  the  quality  of  GRADES. 
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1 .  Grade  Instructions  for  All  Participants 

Please  complete  the  GRADE  every  20  minutes  after  STARTEX  until 
FINEX  according  to  the  following  instructions. 

1 .  Do  not  look  at  any  display  while  completing  GRADE. 

2.  Record  your  duty  position  (e.g.,  Air  Boss,  Redl,  etc.)  and  the 

current  local  time  (using  GPS  or  cell  phone)  in  the  upper  left 
portion  of  the  map. 

3.  Use  the  blue  marker  to  indicate  where  you  believe  the  Blue  forces 
are  located; 

4.  Use  the  blue  marker  to  identify  who  you  think  it  is.  If  you  are 
unsure  of  which  Blue  force  it  is,  put  a  question  mark  next  to  the 
location  mark. 

5.  Draw  a  circle  around  each  Blue  entity  to  indicate  your 
confidence/certaintv  that  they  are  located  where  you  indicated.  A 
small  circle  indicates  high  confidence/certainty.  A  large  circle 
indicates  low  confidence/certainty. 

6.  Follow  the  same  procedure  for  Red  Forces  with  your  red  marker. 

7.  Use  the  red  marker  to  indicate  where  you  believe  the  Red  forces 
are  located  and  then  use  the  red  marker  to  identify  who  you  think 
it  is.  If  you  are  unsure  of  which  red  force  it  is  then  use  a  question 
mark  to  identify  the  location  mark.  Draw  a  circle  around  each  red 
entity  to  indicate  your  confidence/certainty  that  they  are  located 
where  you  indicated.  A  small  circle  indicates  high  confidence/ 
certainty.  A  large  circle  indicates  low  confidence/certainty. 

GRADE  training.  Training  will  reduce  inconsistencies  in  how  participants 
complete  GRADES.  Creating  a  training  folder  is  a  solution  that  should  facilitate 
training  in  as  short  a  time  as  possible.  The  training  folder  could  consist  of: 
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1 .  A  three  ring  binder. 

2.  Accurate  geo-referenced  maps  of  the  area  of  operations. 

3.  Opposite  the  GRADE  map  should  be  a  page  with  instructions 
indicating  what  needs  to  be  done  to  complete  each  practice 
GRADE  (GRADES  should  be  completed  until  there  are  no  errors). 

4.  A  blue  and  a  red  permanent  marker  should  be  provided. 

The  instructions  for  participants  to  follow  to  complete  the  practice  GRADE 
could  include  the  following: 

1.  Your  position  is  Redl. 

2.  The  time  is  09:20. 

3.  Indicate  the  placement  of  Blue2  anywhere  on  the  map  and  indicate 
that  you  are  highly  confident  in  that  location. 

4.  Indicate  the  placement  of  Red3  anywhere  on  the  map  and  indicate 
you  have  low  confidence  in  that  location. 

5.  Indicate  the  placement  of  a  Blue  force  on  the  map  that  you  cannot 
specifically  identify  and  indicate  you  are  highly  confident  of  its 
location. 

Researchers  should  generate  several  versions  of  the  practice  instructions 
varying  time  and  Blue/Red  force  identity. 

Recommended  procedures  to  follow  for  training  are  as  follows: 

1.  Have  the  participant  read  the  GRADE  instructions.  Answer  any 
questions  the  participant  may  have. 

2.  Flip  to  the  first  GRADE  practice  problem  and  have  the  participant 
read  the  instructions  and  then  complete  the  practice  GRADE. 

3.  Review  the  Practice  GRADE  for  errors  and  have  the  participant 
complete  another  practice  GRADE.  Repeat  until  the  participant  has 
completed  three  in  a  row  correctly. 
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Using  this  method  to  complete  GRADE  training  should  ensure  correctly 
completed  measures  and  usable  GRADE  data  returns. 

Quality  control  checks  for  GRADE.  Quality  control  checks  should  occur 
immediately  after  each  practice  GRADE  and  after  the  first  actual  GRADE  is 
completed.  If  the  suggested  instructions  and  training  are  used,  quality  control 
checks  may  need  to  be  conducted  when  the  participants  turn  in  their  completed 
GRADES. 

C.  FUTURE  FOR  GRADE 

One  possible  solution  to  reduce  data  collection  error  is  to  create  a  digital 
or  automated  GRADE.  This  will  reduce  the  overhead  of  supplies  needed  to 
complete  the  GRADE.  A  digital  GRADE  could  also  provide  the  capability  of 
forced  choice  responses  thus  removing  a  major  contributor  to  the  error  in  the 
data  collected  for  TNT  07-03.  Furthermore,  a  digital  GRADE  will  ensure 
standardization  in  how  participants  complete  the  GRADE  and  will  provide  a 
consistent  temporal  referent.  The  proposed  computerized  GRADE  should  have 
the  following  capabilities: 

1.  File  Manipulation  Capabilities 

•  Import  geographic  maps  with  specific  UTM,  Lat/Long  or  MGRS  ranges. 

.  Save  and  load  the  currently  annotated  geographic  map. 

•  Export  user  annotations  and  the  result  of  disparity  analysis  to  a 
commercially  available  spreadsheet  software  (e.g.,  MS  Excel)  for 
tabulation  and  statistical  analysis. 

.  Import  user  defined  symbols  to  complement  MIL-STD  2525B  symbols 
(i.e.,  common  warfighting  symbology)  and  provide  them  in  the 
annotation  tool  bar. 

2.  Annotation  Capabilities 

•  Capability  to  indicate  position  and  strength  of  friendly,  enemy  and  other 
individual  and  unit  icons  on  the  map. 
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.  Software  library  of  icons  (including  MIL-STD  2525B)  and  the  capability 
to  create  and  add  new  icons. 

•  ‘Click  and  drag-able’  capability  for  icons. 

•  Ability  to  place  confidence  or  certainty  circles  around  icons. 

•  Ability  for  user  voice-over  narration  while  drawing. 

•  Ability  to  draw  battlefield  graphics  (e.g.,  avenues  of  approach  and  axis 
of  advance). 

•  Ability  to  collect  and  distinguish  “ground  truth”  from  perceived  or 
predicted  truth. 

3.  Analysis  Capabilities 

•  Real  time  analysis  of  current  data. 

•  Compute  disparity  between  ground  truth  and  reported  conditions. 


D.  FUTURE  STRATEGIES  FOR  POPULATING  DMSC 

Researchers  need  to  reevaluate  the  current  Ground  Truth  data  collection 
strategy.  Researchers  are  using  a  process  tracing  method  to  examine  the 
effects  of  task,  environmental,  and  individual  difference  factors  over  the  course  of 
a  two  hour  scenario  to  make  inferences  about  decision  strategies  and  the 
situated  cognition  that  may  have  contributed  to  choosing  a  course  of  action.  To 
accomplish  this  goal,  researchers  try  to  collect  as  much  data  as  possible  to  build 
an  accurate  representation  of  what  occurred  during  a  two  hour  scenario.  This 
large  amount  of  data  must  be  collected  from  multiple  sources  and  time  stamped. 
It  is  critical  to  have  a  single  temporal  referent  when  combining  multiple  data 
sources  to  achieve  an  accurate  representation  of  what  occurred,  more  over  with 
out  a  single  temporal  referent  it  may  become  impossible  to  combine  multiple  data 
sources. 

One  data  source  that  needs  to  be  changed  is  the  current  method  used  to 
populate  Oval  3.  With  the  benefit  of  hindsight,  it  is  apparent  that  more  frequent 
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screen  captures  would  have  been  better  than  once  every  fifteen  minutes.  A 
better  method  to  collect  data  for  Oval  3  would  be  to  video  tape  the  display  of 
interest  and  store  this  video  recording  on  a  hard  drive.  Storing  the  footage  on  a 
hard  drive  would  remove  the  step  of  converting  video  tape  storage  to  hard  drive 
storage  for  later  analysis. 

Rather  than  trying  to  reconstruct  an  entire  two  hour  scenario,  it  may  be 
better  to  identify  key  events  of  interest  and  then  perform  process  traces  on  these 
events.  Smaller  vignettes  require  less  data  to  populate  the  DMSC  and  are  easier 
to  analyze.  Then,  over  time  as  data  collection  processes  and  procedures  are 
refined,  researchers  can  attempt  to  capture  entire  scenarios.  Researchers 
should  collaborate  with  TNT  planners  to  include  smaller  or  more  focused 
vignettes  for  data  collection  opportunities.  These  vignettes  would  maximize 
researchers’  efforts  to  collect  data  and  ensure  that  the  right  data  are  collected  to 
allow  a  process  trace  of  events  of  interest  through  the  DMSC.  The  vignette  data 
collection  efforts  would  be  a  useful  step  to  demonstrate  the  efficacy  of  GRADE 
as  a  method  and  process  by  which  the  DMSC  model  can  be  validated  and 
applied  for  field  research  of  NCW  systems. 

The  GRADE  continues  to  demonstrate  potential  to  capture  situated 
cognition  not  only  in  laboratory  settings,  but  also  in  field  settings.  Following  the 
recommendations  in  this  thesis  it  will  be  possible  to  capture  Ovals  4  and  5  in 
future  research.  This  provides  the  ability  to  populate  all  the  Ovals  of  the  DMSC 
model.  The  DMSC  then  could  be  used  to  empirically  test  different  NCW 
systems,  thus  providing  decision-makers  a  more  thorough  understanding  of  the 
capabilities  of  each  NCW  systems. 
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APPENDIX 


GRADE 


Instructions  for  GRADE 


(iff  eographical  ecall  and  Analysis  of  Data  in  the  Environment) 


Please  complete  the  GRADE  every  20  minutes  after  STAR  TEX  until  FINEX  according  to  the 
following  instructions. 

1 .  Turn  away  from  yo  u  r  i  nf  o  rmati  o  n  d  isp  I  ay,  m  o  nito  r,  etc. 

2.  Record  your  duty  position  (e.g..  Air  Boss,  Red  1,  etc.)  and  the  current  time  in  the  upper  left 
portion  of  the  map. 

3.  Use  the  blue  marker  te  identify  where  you  believe  the  blue  forces  are  located.  Draw  a 
circle  around  each  blue  entity  to  indicate  your  confidence/certainty.  A  small  circle  indicates 
high  conftdencefceriainfy .  A  large  circle  indicates  low  conftdencefcerfainfy . 

4.  Use  the  red  marker  te  identify  where  you  believe  the  red  forces  are  located.  Draw  a  circle 
around  each  red  entity  to  indicate  your  confidence/certainty.  A  small  cimie  indicates  high 
conftdencefcerfainfy.  A  large  circle  indicates  low  conftdencefcerfainfy. 
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